Inter-Rater Agreement in the Assessment of Video Recordings of Eye Drop Instillation by Glaucoma Patients

PURPOSE To create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders. DESIGN Prospective cross-sectional study. PARTICIPANTS 78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications. METHODS Participants were video recorded while self-instilling artificial tears sequentially to both eyes. Three masked observers graded these video recordings on three criteria: efficacy (the determination of whether an eye drop was instilled on the ocular surface), safety (assessment of whether the tip of the medication bottle made contact with the ocular surface or eyelids), and efficiency (the number of eye drops expressed from the bottle). MAIN OUTCOME MEASURES After grading the video recordings based on efficacy, safety, and efficiency, kappa statistics were used to estimate inter-rater agreement. RESULTS The mean kappa level of agreement for efficacy, safety, and efficiency was 0.64 (95% confidence interval (CI), 0.42-0.87), 0.73 (95% CI, 0.58-0.88), and 0.62 (95% CI, 0.42-0.81), respectively. CONCLUSIONS We demonstrated good inter-rater reproducibility of the masked analysis of video recordings of patients self-instilling eye drops based on three criteria: efficiency, safety, and efficacy.